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Abstract 

In  fast-paced,  distributed,  multi-echelon,  collaborative  environments  such  as  those 
experienced  by  the  United  States  military  during  Operations  Enduring  Freedom  and  Iraqi 
Freedom,  task  performance  very  often  depends  on  effective  and  efficient  information 
exchange.  One  way  to  facilitate  information  exchange  is  to  ensure  that  information 
producers  and  consumers  develop  a  thorough  shared  “situation  context.”  Unfortunately, 
developing  such  a  shared  understanding  of  situation  context  is  extremely  difficult  in  these 
types  of  environments.  This  paper  discusses  a  series  of  experiments  that  investigated 
methods  of  communicating  context  to  information  producers,  and  the  impact  this  had  on 
task  performance.  The  results  of  these  studies  are  intended  to  facilitate  the  design  and 
development  of  tools  to  aid  information  producers  and  consumers  in  quickly  and  easily 
building  shared  situation  context,  and  developing  useful,  usable  information  products 
based  on  this  shared  understanding.  This  paper  reports  the  methods  and  results  of  the 
experiments  conducted.  The  experiment  results  highlight  important  issues  associated  with 
building  a  shared  situation  context  among  information  producers  and  consumers,  and 
provide  important  insights  necessary  to  develop  tools  to  facilitate  efficient  and  effective 
information  exchange. 

Introduction 

A  fundamental  aspect  of  Network-centric  command  decision-making  is  the  exchange  of 
information  so  as  to  share  knowledge  and  support  decision  making.  In  decision-making 
environments,  information  is  often  being  exchanged  among  individuals  and  teams  at 
different  echelons,  working  on  different  schedules  in  distributed  global  locations.  In  such 
an  environment,  information  producers  will  have  varying  degrees  of  knowledge  about  the 
people  with  whom  they  are  sharing  information,  and  this  can  adversely  affect  information 
exchange.  Conventional  wisdom  would  suggest  that  we  simply  build  aids  to  assist  people 
in  understanding  each  others’  needs,  yet  as  scientists  and  technologists  we  have  a  poor 
understanding  of  which  aspects  of  knowledge  about  the  people  with  whom  one  is 


collaborating  impact  the  effectiveness  of  sharing  information.  This  issue  is  becoming 
more  critical  as  the  military  increasingly  moves  to  a  network-centric  concept  of 
operations.  Operations  in  modem  distributed,  multi-echelon  environments  are 
increasingly  reliant  on  advanced  collaboration  technologies  which  have  made  it  possible 
to  transfer  large  amounts  of  information  quickly  to  almost  any  location.  However  these 
technologies  do  not  necessarily  facilitate  the  exchange  of  the  right  information.  The 
study  described  here  is  one  of  a  series  being  conducted  to  look  at  strategies  for  taking 
best  advantage  of  the  capabilities  presented  by  network-centric  operations  and  identifying 
strategies  to  mitigate  potential  pitfalls 

In  any  collaborative  environment,  optimal  performance  depends  on  information  exchange 
being  as  effective  and  efficient  as  possible.  Information  exchange,  as  it  is  discussed  here, 
refers  to  the  process  of  producing  information  for,  or  consuming  information  from,  a 
shared  information  space.  Participants  in  the  information  exchange  process  are  therefore, 
producers ,  consumers ,  and  often,  both.  Information  exchange  is  effective  if  producers  are 
creating  all  of  the  information  that  consumers  need  and  consumers  are  accessing  all  the 
information  they  need,  i.e., 

Effectiveness  =  Relevant  information  produced  and  consumed 

All  relevant  information  in  the  information  space 

Information  exchange  is  efficient  if  producers  are  not  creating  information  that  is 
extraneous  to  the  consumers’  tasks,  i.e., 

Efficiency  =  Irrelevant  information  produced  and  consumed 

All  irrelevant  information  in  the  information  space 


Information  Exchange  Problems 

Based  on  previous  cognitive  task  analyses  and  evaluations  of  collaboration  tool  use  in 
distributed  multi-echelon  environments  (e.g.,  Moore  &  Averett,  1999;  Oonk,  Smallman, 
Moore,  &  Morrison,  2001;  Rogers,  Oonk,  Moore,  &  Morrison,  2002;  Smallman,  Oonk,  & 
Moore,  2001)  we  have  identified  two  problems  that  contribute  to  inefficient  and 
ineffective  information  exchange: 

1.  Poor  understanding  of  consumer  information:  Tasks  and  information 
requirements.  Military  decision  makers  and  their  staffs  consistently  state  that 
understanding  each  other’s  tasks,  intentions,  priorities,  and  information 
requirements  are  important  but  that  this  understanding  does  not  always  exist. 
Information  producers  often  do  not  understand  what  information  the  consumer 
needs  and  how  he  will  be  using  that  information. 

2.  Requirement  for  tailored  information.  Military  decision  makers  have  also 
indicated  the  need  for  information  products  that  are  customized  to  meet  their  own 
specific  needs.  Such  customization  is  not  so  easily  accomplished  in  a  multi- 
echelon/multi-consumer  environment.  When  the  production  of  customized 
information  has  been  available,  it  has  frequently  been  created  with  the  requirements 
of  a  single  consumer  in  mind.  However,  other  consumers  are  often  required  to 
access  and  use  the  product  even  though  it  has  not  been  tailored  to  these  consumers’ 
needs. 


The  identification  of  these  issues  underscores  an  important  requirement  in  information 
exchange  environments:  communication  of  context.  In  order  to  alleviate  such  problems,  it 
is  necessary  to  provide  producers  with  improved,  shared  understanding  of  “situation 
context,”  with  respect  to  those  consumers  for  whom  they  create  information.  This  paper 
discusses  a  series  of  experiments  in  which  we  investigated  methods  of  communicating 
situation  context  to  information  producers.  Specifically,  previous  and  ongoing 
experiments  and  evaluations  have  examined  1)  what  information  should  be  displayed, 
and  2)  how  best  to  display  that  information. 

The  importance  of  context. 

The  understanding  of  context  by  individuals  is  related  to  the  state  of  their  evolving 
mental  models.  Mental  models  are  internal  knowledge  structures  held  by  individuals  that 
allow  them  to  make  sense  of  their  surroundings  (Klimoski  &  Mohammed,  1994).  Mental 
models  provide  individuals  with  a  context  within  which  to  make  inferences  about, 
understand  relationships  between  and  attribute  meaning  to  events  in  their  environment,  as 
well  as  make  predictions  about  the  future  (Rouse  &  Morris,  1986).  Recently,  the  idea  of 
“shared  mental  models”  -  such  representations  shared  across  members  of  a  group  of 
individuals  -  has  been  used  to  help  explain  team  behaviors  and  performance  (e.g., 
Cannon-Bowers,  Salas  &  Converse,  1993,  Mathieu,  Goodwin,  Heffner,  Salas,  &  Cannon- 
Bowers,  2000,  Salas,  Cannon-Bowers,  &  Blickensderfer,  1997).  The  degree  to  which 
these  models  are  shared  is  believed  to  be  linked  to  team  performance,  team  effectiveness, 
adaptation,  and  coordination  (Cannon-Bowers,  et  al.,  1993).  Because  appropriate  mental 
models  enable  team  members  to  form  accurate  explanations  and  expectations  about  the 
task,  the  environment,  and  each  other,  the  quantity  and  quality  of  information  exchange 
should  change  with  the  quality  of  the  mental  model  (and  vice  versa)  (Caldwell  & 
Everhart,  1998;  Mathieu  et  al.,  2000). 

Communicating  context. 

Unlike  most  studies  in  the  literature  that  measure  the  overlap  between  team  members’ 
mental  models  and  their  relationship  to  team  performance,  our  research  attempts  to 
manipulate  mental  models  and  look  at  the  effect  on  team  performance.  We  do  this  in 
order  to  understand  the  relationship  between  shared  mental  models  and  the  effectiveness 
and  efficiency  of  information  exchange  (Oonk,  Schermerhom,  Glaser,  &  Manes,  2003). 

In  order  to  manipulate  mental  models  in  an  experimental  setting,  one  must  have  an 
understanding  of  what  aspects  of  the  mental  model  are  important  -  in  other  words,  what 
aspects  of  the  mental  model  should  be  shared  and  communicated  to  team  members. 
Mathieu  et  al.  (2000)  identified  three  categories  of  mental  models:  1)  Shared 
understanding  of  technology  used  by  team  members,  its  functioning  and  control  and  how 
other  team  members  interact  with  it;  2)  Shared  understanding  of  the  team  task,  strategies 
used  to  accomplish  it,  environmental  conditions,  and  potential  problems;  3)  Shared 
understanding  of  interaction  between  team  members,  information  flow,  roles  and  role 
relationships,  information  sources  and  knowledge,  and  skills  of  team  members.  This  third 
type  of  mental  model  comprises  the  features  that  we  believe  define  “ situation  context.  ” 

The  experiments  described  in  this  paper  investigated  the  effects  of  presenting  different 
aspects  of  situation  context  to  information  producers.  Based  on  previous  analyses  and 


discussions  with  information  producers  and  consumers,  we  identified  four  components  of 
situation  context  that  could  potentially  be  communicated. 

•  Mission  (What  the  organization  is  doing)-.  Many  times  information  producers  are 
only  aware  of  the  general  context  under  which  their  team  or  organization  is 
operating.  They  understand  the  mission  or  high-level  goals  of  the  organization  but 
not  the  specific  roles  that  the  consumers  with  whom  they  share  information  play 
within  this  mission.  Without  this  knowledge,  producers  must  make  assumptions, 
based  on  prior  experience,  about  what  consumers  are  doing  and  what  information 
they  need  to  support  them. 

•  Stated  information  requirements  (What  I  need):  Information  consumers  may 
provide  producers  with  a  list  of  the  information  requirements  that  consumers 
believe  they  need  to  conduct  their  tasks.  This  gives  producers  more  specific 
information  about  consumers’  needs  than  just  the  mission  information.  However,  a 
problem  with  the  use  of  such  a  list  is  that  it  assumes  that  consumers  know 
everything  that  they  need  (as  well  as  what  producers  are  able  to  provide  them)  and 
are  able  to  communicate  that  information.  Many  times  producers  have  information 
that  consumers  need  of  which  they  are  not  aware.  Producers  are  unlikely  to  share 
such  information  if  they  don’t  know  why  consumers  need  it. 

•  Current  task  (What  I’m  working  on):  Information  consumers  may  provide 
producers  with  a  list  of  the  current  tasks  on  which  they  are  working.  A  benefit  of 
presenting  task  to  producers  is  that  they  will  be  more  likely  to  share  information 
with  consumers  that  they  believe  will  be  useful  to  their  tasks,  whether  it  was 
explicitly  asked  for  or  not.  However,  this  assumes  that  producers  have  a  certain 
level  of  knowledge  about  what  the  tasks  entail  and  what  information  is  necessary  to 
support  them. 

•  Information  requirements  and  task  (What  1  need  and  what  I’m  working  on).  Based 
on  the  limitations  of  the  methods  listed  above,  the  intuitively  optimal  method  is  to 
communicate  both  consumers’  tasks  and  information  requirements.  This  would 
make  it  more  likely  that  information  that  consumers  believe  that  they  need  and  that 
producers  believe  the  consumers  need  would  be  shared.  However,  this  amount  of 
information  may  be  unnecessary  and  it  may  even  be  associated  with  increased 
workload  for  producers. 

In  two  experiments,  we  examined  how  well  each  of  the  above  methods  of  communicating 
consumer  information  supported  information  production.  Participants  played  the  role  of 
an  information  producer  in  a  multi-echelon,  multi-consumer  military  organization.  After 
reviewing  a  set  of  information  items,  they  were  given  information  about  the  consumer  - 
either  1)  the  overall  mission,  2)  the  mission  and  the  consumer’s  stated  information 
requirements,  3)  the  mission  and  the  consumer’s  tasks,  or  4)  the  mission,  the  consumer’s 
information  requirements,  and  tasks.  Based  on  this  information,  participants  sorted  each 
of  the  information  items  based  on  how  relevant  they  believed  its  content  was  to  the 
consumer  (choices  were  relevant,  somewhat  relevant,  or  not  relevant).  In  each 
experiment,  participants  performed  the  sort  task  twice,  using  two  independent  sets  of 
information  items.  In  Experiment  1,  they  sorted  the  items  for  a  different  consumer  each 
time.  In  Experiment  2,  they  sorted  the  items  for  one  consumer  in  one  block,  and  for  two 
consumers  in  the  other.  Participants  were  given  the  consumer  information  for  both 


consumers  in  the  two  consumer  block  and  they  were  instructed  to  perform  the  sort  task  as 
if  they  were  creating  an  information  product  for  both  consumers.  This  manipulation 
allowed  us  to  examine  how  useful  an  information  product  created  for  multiple-consumers 
would  be  for  each  of  the  individual  consumers. 

We  focused  on  participants’  performance,  in  terms  of  effectiveness  and  efficiency.  We 
define  effectiveness  as  how  well  the  participants  performed  at  deciding  which 
information  was  relevant.  We  define  efficiency  as  how  well  the  participants  performed  at 
deciding  which  information  was  not  relevant.  Performance  was  based  on  the  overlap 
between  participants’  decisions  and  ratings  provided  by  military  subject  matter  experts 
(SMEs)  who  had  assumed  the  role  of  the  consumer  whom  participants  were  sorting  for. 
We  used  both  accuracy  (proportion  correct)  and  signal  detection  measures  (sensitivity 
and  response  bias)  to  define  performance.  In  order  to  look  at  the  effects  of  tailoring 
information  (i.e.,  the  cost  to  one  consumer  accessing  information  products  created  with 
another  consumer’s  information  needs  in  mind),  we  also  looked  at  the  overlap  between 
participants’  decisions  and  ratings  provided  by  SMEs  who  had  assumed  the  role  of  the 
consumer  whom  the  participants  were  not  sorting  for. 

Experiment  1 

Method 

Participants.  Thirty-two  participants  each  served  in  a  single  2-hour  session.  Most  ( n  = 

24)  were  students  from  a  local  university  or  members  of  the  community  who  were  paid 
for  their  participation.  The  rest  were  employees  of  the  facility  at  which  data  were 
collected. 

Stimuli.  Two  sets  of  40  information  items,  Scenarios  A  and  B,  were  compiled.  The  items 
contained  content  related  to  various  degrees  to  an  overall  Humanitarian  Assistance  - 
Disaster  Relief  (HA-DR)  mission.  They  consisted  of  pictures,  text  information,  tables, 
weather  and  timeline  charts,  and  tactical  or  other  maps.  Several  handouts  were  also 
created  for  use  as  visual  aids.  These  were:  1)  a  text  copy  of  the  scenario  and  map 
highlighting  scenario  locations;  2)  a  chart  of  the  organizational  structure  and  information 
flow  between  consumers  and  producers;  and  3)  consumer  information  handouts  (one  for 
each  consumer  in  each  display  condition)  that  were  used  to  present  information  about  the 
consumer  in  the  form  of  bulleted  lists. 

Procedure.  At  the  beginning  of  the  experiment,  participants  were  introduced  to  the 
organization  and  HA-DR  scenario.  They  were  told  that  they  were  part  of  a  multi-level 
military  organization  consisting  of  information  consumers  and  producers  conducting  a 
fictional  mission.  Appropriate  visual  aids  were  provided.  They  were  told  that  they  would 
be  playing  the  role  of  one  of  the  information  producers. 

Each  experimental  block  consisted  of  two  phases:  a  “Review”  Phase  and  a  “Sort”  Phase. 
During  the  Review  Phase,  participants  were  given  25  minutes  to  view  the  information 
items  presented  in  a  list.  Participants  were  told  to  spend  the  time  becoming  familiar  with 
the  content  of  the  information  items  and  gain  situation  awareness  about  the  mission.  To 
review  an  item,  participants  double-clicked  on  it  with  the  mouse.  They  could  view  the 


items  as  many  times  and  for  as  long  as  they  wished  but  were  encouraged  to  attempt  to 
view  all  items  in  the  list  in  the  given  time. 

Following  the  Review  Phase,  participants  were  given  information  about  the  consumer  for 
whom  they  would  be  sorting  information.  This  varied,  dependent  on  which  of  the  four 
“consumer  information”  display  conditions  participants  were  assigned  to.  These  display 
conditions  were: 

•  Mission  Only  (M  Only):  participants  received  information  about  the  overall 
mission  being  conducted  by  their  organization. 

•  Mis sion+ Information  (M+I):  in  addition  to  the  mission  information,  participants 
were  provided  a  list  of  information  that  they  were  told  the  consumer  stated  that  they 
needed. 

•  Mission+Task  (M+T):  in  addition  to  the  mission  information,  participants  were 
provided  a  list  of  tasks  that  they  were  told  the  consumer  was  currently  working  on. 

•  Mission+ Information +Task  ( M+I+T '):  in  addition  to  the  mission  information, 
participants  were  provided  a  list  of  information  requirements  and  a  list  of  the 
consumer’s  current  tasks. 

All  participants  first  read  information  about  the  mission.  They  were  then  told  which 
consumer  (either  the  Commander,  Joint  Task  Force  (CJTF)  or  his  Deputy  Operations 
Officer  (DOO)),  they  would  be  sorting  information  for  in  that  experimental  block. 
Participants  in  the  M  Only  condition  then  began  the  Sort  Phase  of  the  experiment. 
Participants  in  the  other  3  conditions  were  given  additional  information  about  the 
consumer  before  beginning  the  Sort  Phase. 

During  the  Sort  Phase,  participants  sorted  each  information  item  according  to  whether 
they  believed  the  content  of  the  information  item  was  “relevant,”  “not  relevant,”  or 
“somewhat  relevant”  based  on  what  they  understood  about  the  consumer’s  specific  task. 
It  was  stressed  to  participants  that  some  information  was  relevant  to  the  overall  HA-DR 
mission  but  not  necessarily  to  the  consumer’s  specific  task.  Participants  sorted  the 
information  by  dragging  and  dropping  the  items  with  the  mouse  into  one  of  three 
windows,  labeled  “Relevant”,  “Somewhat  Relevant,”  or  “Not  Relevant,”  on  the  right  side 
of  the  data  collection  interface.  Participants  were  given  15  minutes  to  perform  the  sort 
task. 

Design.  Eight  participants  served  in  each  of  the  four  consumer  information  display 
conditions  (M  Only,  M+I,  M+T,  M+I+T).  Each  participant  served  in  two  separate  blocks 
and  in  each  block  they  reviewed  information  from  a  different  information  set  (Scenarios 
A  or  B)  and  sorted  information  for  a  different  consumer  (the  CJTF  or  DOO).  The  pairings 
between  consumer  and  information  set  (e.g.,  sorting  Scenario  A  for  the  CJTF)  and  the 
order  of  blocks  were  counterbalanced. 

In  order  to  examine  the  effects  of  tailored  content,  i.e.,  how  useful  an  information  product 
created  for  one  consumer  is  when  it  is  accessed  by  another  consumer,  the  experimental 
design  included  another  variable:  correspondence.  We  compared  performance  across  two 
levels  of  correspondence:  in  the  Match  condition,  we  based  the  performance  measures  on 


the  ratings  made  by  the  same  consumers  that  participants  believed  they  were  sorting  for 
(e.g.,  the  CJTF  ratings  for  the  CJTF  consumer  condition);  in  the  Mismatch  condition  we 
based  them  on  the  ratings  made  by  the  other  consumer  (e.g.,  the  DOO  ratings  for  the 
CJTF  consumer  condition). 

Ratings.  Five  SMEs  served  as  raters.  All  of  them  had  previously  served  in  the  military 
(mean  years  of  service  =  18.1,  SD  =  8.6,  Range  =  9.5  -  30)  and  were  currently 
professionals  working  in  military  contexts  (mean  years  of  service  =  10.8,  SD  =  5.3, 
Range  =  5  -  19).  Ratings  were  collected  from  the  raters  with  the  same  Review/Sort  task 
and  tool  used  by  the  experimental  participants.  SMEs  were  given  the  information 
provided  in  the  M+I+T  condition,  i.e.,  all  information  about  the  consumer  whose  role 
they  were  asked  to  assume. 1[1] 

Results  and  Discussion 

Three  performance  measures  were  analyzed,  based  on  participants’  degree  of 
concordance  with  the  SME  ratings  of  the  relevance  of  information  items.  These  were:  1) 
Accuracy:  Proportion  correct  for  all  items  and  for  each  of  the  three  rating/sorting 
categories,  e.g.,  proportion  of  items  rated  as  relevant  by  the  consumers  that  were  sorted  as 
relevant  by  participants;  and  two  signal  detection  measures  based  on  each  participant’s 
mean  hit  and  false  alarm  rates  for  all  items  and  for  each  of  the  three  rating/sorting 
categories:  2)  Sensitivity  -  d’  values;  3)  Response  bias  -  j3  values.  After  analyzing 
performance  over  all  items,  we  analyzed  Relevant  and  Not  Relevant  items  separately. 
These  last  analyses  allowed  us  to  evaluate  participants’  ability  to  make  two  important  and 
complementary  types  of  decisions  that  relate  to: 

1.  Effectiveness:  giving  consumers  access  to  useful  content  (Relevant  items) 

2.  Efficiency:  not  giving  access  to  content  that  is  not  useful  (Not  Relevant  items) 

Overall  Performance.  Not  surprisingly,  communicating  situation  context  related  to  the 
tasks  and  information  requirements  of  a  specific  consumer  allows  producers  to  create  an 
information  product  that  meets  that  consumer’s  needs.  However,  this  product  will  be  less 
useful  to  other  consumers,  who  are  working  on  different  tasks  and  have  different 
requirements.  As  expected,  performance,  as  measured  by  both  accuracy  (proportion 
correct)  and  sensitivity  id'),  was  higher  when  it  was  based  on  the  ratings  made  by  the 
consumer  that  was  being  sorted  for  (Match  condition)  than  when  based  on  the  other 
consumer’s  ratings  (Mismatch)  (.66  vs  .56,  F(l,  28)  =  40.39,  p  <  .0001  for  accuracy;  1.33 
vs.  .94,  F(l,  28)  =  34.15,  p  <  .0001  for  sensitivity). 

Deciding  what  is  relevant:  effectiveness  of  information  exchange:  Mean  accuracy, 
sensitivity,  and  response  bias  for  Relevant  items  are  shown  in  Figure  1.  The  results 
indicate  that  task  information  is  the  most  important  aspect  (of  those  examined)  of 
situation  context  in  supporting  effective  information  exchange.  Producers  who  know  the 
consumer’s  task  perform  better  when  deciding  what  is  relevant  to  the  consumer  than 


1(11  Because  we  base  performance  on  the  SME  ratings  (as  the  “true”  ratings),  items  rated  as  “relevant”,  “not 
relevant”  and  “somewhat  relevant”  are  henceforth  referred  to  as  “Relevant  items”,  “Not  Relevant  items”, 
and  “Somewhat  Relevant  items”,  respectively. 


those  who  only  know  the  overall  mission  or  are  provided  with  a  list  of  information 
requirements. 


Correspondence 


Figure  1.  Mean  accuracy  (top),  sensitivity  (bottom  left)  and  response  bias  (bottom  right)  for 
Relevant  items  in  Experiment  1.  Note,  M  =  Mission,  I  =  Information  Requirements,  T  =  Task. 


Performance  was  better  in  the  Match  condition  than  Mismatch  condition,  as  measured  by 
sensitivity  (F(l,  28)  =  31.01,  p  <  .0001),  and  accuracy  (F(l,  28)  =  26.03,  p  <  .0001.)  In 
the  Match  condition,  participants  who  were  given  information  about  the  consumer’s  task 
and/or  information  requirements  performed  better  when  deciding  whether  an  information 
item  was  relevant  to  the  consumer’s  task  than  those  who  were  given  just  the  overall 
mission  information.  Sensitivity  values  (d’s)  were  highest  with  the  M+T  and  M+I 
displays  and  lowest  with  the  M  Only  display  (although  this  difference  was  not 
statistically  significant,  F( 3,  28)  =  1.07,  p  =  .37).  The  reverse  pattern  was  found  for  the 
Mismatch  condition:  participants  in  the  M  Only  conditions  were  more  accurate  (F( 3,  28) 
=  5.10,  p  =  .01)  and  exhibited  higher  d’  values  than  those  in  the  other  conditions 
(although  this  difference  was  not  statistically  significant,  F( 3,  28)  =  1 .50,  p  =  .24).  This 
suggests  that,  if  an  information  producer  is  tailoring  an  information  product  for  a 
particular  consumer,  information  exchange  will  be  more  effective  with  respect  to  that 
consumer  (i.e.,  they  will  be  more  likely  to  get  the  information  that  they  need)  but  at  a 
cost  to  the  other  consumers.  Information  exchange  would  actually  be  more  ineffective 
with  respect  to  other  consumers,  i.e.,  they  would  be  less  likely  to  get  the  information  they 
need  to  perform  their  task. 


Deciding  what  is  not  relevant:  efficiency  of  information  exchange:  Mean  accuracy, 
sensitivity,  and  response  bias  for  Not  Relevant  items  are  shown  in  Figure  2.  The  results 
indicate  that  task  information  is  also  the  most  important  aspect  (of  those  examined)  of 
situation  context  in  supporting  efficient  information  exchange. 
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Figure  2.  Mean  accuracy  (top),  sensitivity  (bottom  left)  and  response  bias  (bottom  right)  for  Not 
Relevant  items  in  Experiment  1.  Note,  M  =  Mission,  I  =  Information  Requirements,  T  =  Task. 


Participants  who  were  given  information  about  the  consumer’s  task  performed  better 
when  deciding  whether  an  information  item  was  not  relevant  to  the  consumer’s  task  than 
those  who  were  given  just  the  overall  mission  information  or  the  mission  information 
plus  the  consumer’s  information  requirements  (for  accuracy,  F(3,  28)  =  24.33,  p  <  .0001). 
There  was  no  difference  in  performance  across  display  conditions  in  the  Mismatch 
condition  This  suggests  that,  if  an  information  producer  is  tailoring  an  information 
product  for  a  particular  consumer  based  on  their  understanding  of  the  consumer’s  task, 
information  exchange  will  be  more  efficient  for  that  consumer  (i.e.,  it  will  be  less  likely 
that  that  consumer  will  be  given  access  to  irrelevant  information),  with  no  cost  to  the 
other  consumers. 

Experiment  2 

Although  the  results  of  experiment  1  revealed  patterns  of  performance  as  a  function  of 
display  condition  that  were  similar  across  the  different  analyses,  the  results  were  not  very 
robust  (statistically).  This  may  have  been  due  to  a  lack  of  understanding  of  or  focus  on 
the  part  of  participants  on  their  role  as  an  information  producer,  i.e.,  that  the  outcome  of 
their  sorting  behavior  would  be  an  information  product  used  by  the  consumer.  The 
purpose  of  Experiment  2  was  twofold:  1)  We  wanted  to  replicate  the  findings  of 
Experiment  1,  where  participants  sorted  for  a  single  consumer;  2)  We  were  interested  in 
assessing  what  differences  there  might  be  in  information  production  when  producers  had 
multiple  consumers.  The  task  in  the  first  part  of  Experiment  2  was  almost  identical  to  that 
in  the  previous  experiment  except  that  we  changed  the  instructions  slightly,  increasing 
the  emphasis  on  the  participant’s  role  as  an  information  producer  to  obtain  more  powerful 


results.  Specifically,  we  modified  the  operational  definitions  of  “relevant,”  “somewhat 
relevant,”  and  “not  relevant.” 

Second,  we  were  interested  in  the  difference  in  performance  when  producers  had  multiple 
consumers  as  compared  to  one.  Therefore,  in  the  second  part  of  Experiment  2  participants 
were  given  information  about  both  consumers.  The  change  in  role  definitions  described 
below,  which  emphasized  information  production,  facilitated  the  explanation  of  the 
multi-consumer  situation  to  participants  -  they  were  told  to  consider  that  the  information 
product  that  resulted  from  their  sort  would  be  accessible  to  both  consumers. 

Method 

Participants.  Thirty-two  participants  each  served  in  a  single  2-hour  session.  Most  (28) 
were  students  from  a  local  university  or  members  of  the  community  who  were  paid  for 
their  participation.  The  remaining  participants  were  employees  of  the  facility  at  which  the 
data  were  collected. 

Stimuli.  The  stimuli  used  were  the  same  as  in  Experiment  1. 

Procedure.  The  procedure  was  the  same  as  in  Experiment  1,  with  the  following 
exceptions. 

During  training,  the  definitions  of  the  three  sorting  categories  were  changed  slightly  to 
emphasize  the  task  of  information  production.  Changes  were  made  for  two  reasons: 

1.  To  encourage  participants  to  think  about  how  the  outcome  of  the  sort  task  might 
result  in  information  product. 

2.  To  facilitate  the  explanation  to  participants  in  the  Two  Consumer  condition  that 
they  would  be  creating  one  information  product  to  be  used  by  two  people. 

Prior  to  the  beginning  of  the  Sort  Phase,  participants  read  information  about  the  mission. 
They  were  then  told  which  consumer(s)  they  would  be  sorting  information  for  during  that 
experimental  block,  the  CJTF,  DOO,  or  both.  In  blocks  when  participants  sorted  for  two 
consumers  they  were  given  handouts  for  each,  depending  on  the  display  condition  in 
which  they  served. 

Design.  Eight  participants  served  in  each  of  the  four  consumer  information  display 
conditions.  Each  participant  served  in  two  separate  blocks  and,  in  each,  reviewed 
information  from  a  different  information  set.  In  one  block,  participants  sorted  information 
for  one  consumer  (Single  Consumer  condition)  and  in  the  other  they  sorted  for  two 
consumers  (Two  Consumer  condition).  Half  of  the  participants  sorted  information  for  the 
CJTF  in  the  Single  Consumer  condition  and  half  sorted  for  the  DOO.  The  pairings 
between  number  of  consumers  and  information  set  and  the  order  of  blocks  were 
counterbalanced. 

Results  and  Discussion 

One  vs.  Two  Consumers.  The  results  suggest  that  an  information  product  created  based 
on  information  about  more  than  one  decision  maker,  will  be  less  useful  to  each  of  those 
decision  makers  than  would  a  product  created  based  on  each  of  their  own  tasks  and 
information  requirements.  In  general,  accuracy  at  sorting  items  as  relevant  was  higher 
when  participants  sorted  for  one  consumer  (based  on  that  particular  consumer’s  ratings) 


than  when  they  sorted  for  two  consumers  (based  on  each  of  the  two  consumer’s  ratings). 
This  result,  however,  was  only  found  when  the  single  consumer  was  the  CJTF  and  when 
it  was  compared  to  the  two  consumer  condition  based  on  the  DOO  ratings  (post  hoc  p< 
.0125,  F(6,  48)  =  2.41,  p  =  .04  for  the  Number  of  Consumer  X  Consumer  X  Display 
interaction)  .  This  suggests  that  participants  may  have  biased  their  behavior  in  the  two 
consumer  conditions  toward  the  CJTF’s  needs,  regardless  of  whether  the  DOO  would 
also  be  using  the  product  based  on  the  sort.  This  is  not  surprising,  given  that  the  CJTF 
was  at  the  “top”  of  the  organization  and  participants  may  have  felt  his  needs  should  be 
given  higher  weight  than  those  of  his  subordinate.  Sensitivity  was  also  higher  for  sorting 
items  as  relevant  when  participants  sorted  for  one  consumer  ( d ’  =  1.74)  than  when  they 
sorted  for  two  consumers  (1.50  when  based  on  CJTF  ratings,  1.16  when  based  on  DOO 
ratings),  F( 2,  56)  =  3.03,  p  =  .06. 

The  results  also  suggest  that  producers  may  be  more  likely  to  worry  about  “missing” 
important  information  when  creating  an  information  product  for  more  than  one  consumer, 
and  would  be  more  likely  to  provide  information  that  was  not  relevant.  Response  bias 
was  generally  lower  when  sorting  relevant  items  for  two  consumers  than  one  (F( 2,  48)  = 
6.6,  p  <  .01),  but  only  for  participants  that  sorted  for  the  DOO  (F( 2,  48)  =  8.63  for  the 
interaction,  p  <  .01  for  the  ,  post  hoc  p  <  .0125),  indicating  that  participants  were  more 
likely  to  sort  an  item  as  relevant  if  two  consumers  might  access  it.  For  sorting  not 
relevant  items,  response  bias  was  lower  when  sorting  items  for  one  consumer,  but  only 
when  the  two  consumer  performance  was  based  on  DOO  ratings  (F(l,  24)  =  4.52,  p  = 
.04).  The  reversal  in  the  patterns  of  results  for  sorting  the  two  categories  suggests  that 
participants  favored  effectiveness  over  efficiency  in  the  two  consumer  condition. 
Participants  may  be  concerned  about  missing  relevant  information  in  an  information 
product  if  they  believe  more  consumers,  or  just  the  CJTF,  would  have  access  to  it. 

Single  Consumer. 

Overall.  We  again  obtained  results  that  suggest  that  customizing  an  information  product 
based  on  a  specific  consumer’s  information  causes  that  product  to  be  less  useful  to  others, 
who  might  access  it  but  have  different  tasks  and  information  requirements.  As  in 
Experiment  1,  performance  was  better  when  it  was  based  on  the  consumer  for  whom 
participants  were  sorting  than  when  it  was  based  on  the  other  consumer  (F(l,  24)  =  8.31, 
pc. 01). 

Deciding  what  is  relevant:  effectiveness  of  information  exchange:  Mean  accuracy, 
sensitivity,  and  response  bias  for  Relevant  items  are  shown  in  Figure  3.  In  contrast  to  the 
previous  experiment,  where  the  results  suggested  that  task  information  was  the  most 
important  aspect  to  effective  information  production,  the  results  of  Experiment  2  suggest 
that  producers  will  be  most  effective  if  they  are  communicated  the  consumer’s  task  and 
information  requirements.  The  M+I+T  display  supported  the  most  effective  information 
sorting  in  the  Match  condition;  participants  who  were  given  both  the  consumer’s  task  and 
information  requirements  performed  best  when  sorting  items  as  relevant  when  their 
performance  was  based  on  the  consumer  that  they  sorted  for  (F( 3,  24)  =  4.09,  p  =  .02,  for 
accuracy;  F( 3,  24)  =  2.5,  p  =  .08,  for  sensitivity).  However,  as  in  Experiment  1,  there  was 
a  cost  when  performance  was  based  on  the  other  consumer  when  participants  were  given 


consumer  task  and/or  information  requirements  relative  to  just  the  mission  for  sorting 
relevant  items  only  (F( 3,  24)  =  2.6,  p  =  .08,  for  sensitivity)  . 
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Figure  3.  Mean  accuracy  (top),  sensitivity  (bottom  left)  and  response  bias  (bottom  right)  for 
Relevant  items  in  the  Single  Consumer  condition  of  Experiment  2.  Note,  M  =  Mission,  I  = 
Information  Requirements,  T  =  Task. 


Deciding  what  is  not  relevant:  efficiency  of  information  exchange:  Mean  accuracy, 
sensitivity,  and  response  bias  for  Not  Relevant  items  are  shown  in  Figure  4.  The  results 
suggest  that  producers  will  be  most  efficient  if  they  are  communicated  the  consumer’s 
task  and  information  requirements.  The  M+I+T  display  supported  the  most  efficient 
information  sorting  in  the  Match  condition  -  participants  who  were  given  both  the 
consumer’s  task  and  information  requirements  performed  best  when  sorting  items  as  not 
relevant  when  their  performance  was  based  on  the  consumer  that  they  sorted  for 
(although  this  difference  was  not  statistically  significant). 


□  M  only  Dm  ♦  I  Bm  ♦  T  BM  ♦  I  ♦  T 


II  !L In 

o  J _ I _ _ 1 _ L_B^H  Monty  MX  M  +  T  M  ♦  I  ♦  T 


Match 


Morale  h 


Correspondence 


Figure  4.  Mean  accuracy  (top),  sensitivity  (bottom  left)  and  response  bias  (bottom  right)  for  Not 
Relevant  items  in  the  Single  Consumer  condition  of  Experiment  2.  Note,  M  =  Mission,  I  = 
Information  Requirements,  T  =  Task. 


Two  Consumers.  The  results  suggest  that,  when  it  is  necessary  for  producers  to  create  an 
information  product  for  more  than  one  consumer,  the  same  information  about  task  and 
information  requirements  of  all  the  consumers  should  be  provided  to  them.  When 
participants  sorted  information  for  use  by  two  consumers,  patterns  of  performance  as  a 
function  of  display  condition  were  generally  the  same  as  when  they  sorted  for  one 
consumer.  Participants  performed  best  when  sorting  items  as  both  relevant  and  not 
relevant  when  they  were  given  the  consumer’s  mission,  task  and  information 
requirements  (M+T+I)  and  worst  when  they  were  given  the  mission  information  only. 


The  findings  of  Experiment  2  are  important  for  several  reasons: 

1 .  They  replicate  the  results  of  Experiment  1  that  demonstrated  the  cost  of 
customizing  information  products  in  multi-consumer  environments.  Creating  an 
information  product  based  on  knowledge  about  a  single  consumer  will  make  it  more 
useful  to  that  consumer.  However,  other  consumers,  with  different  tasks  and 
information  requirements,  will  find  it  to  be  less  useful  than  a  more  general,  mission- 
oriented  product. 

2.  They  contrast  with  the  results  of  Experiment  1,  where  only  task  information  was 
indicated  as  necessary  to  support  effective  and  efficient  information  exchange,  by 
suggesting  that  producers  should  be  displayed  both  consumer  task  and  consumer 
information  requirements.  This  discrepancy  in  results  is  surprising,  given  that  the 
participants’  task  in  the  Single  Consumer  condition  was  virtually  identical  to  the 
task  in  Experiment  1,  and  they  suggest  that  further  investigation  is  necessary. 

They  extend  the  results  of  Experiment  2  in  a  multi-consumer  situation  and  suggest 
that: 


3. 


a.  Information  products  created  for  more  than  one  consumer  will  be  less  useful  to 
each  of  those  decision  makers  than  would  a  product  created  based  on  each  of 
their  own  tasks  and  information  requirements. 

b.  If  multi-consumer  information  products  are  necessary,  the  tasks  and 
information  requirements  of  all  consumers  should  be  communicated  to  the 
producer. 

General  Discussion 

The  results  suggest  that,  in  order  to  support  the  most  effective  and  efficient  information 
exchange,  producers  should  have  access  to  the  tasks  consumers  are  performing,  and  what 
information  requirements  support  that  task,  in  addition  to  the  general  context  or  mission. 
The  results  highlight  the  importance  of  maintaining  a  shared  understanding  of  the 
situation,  roles,  responsibilities  and  information  requirements  that  is  highlighted  in  the 
shared  mental  model  and  transactive  memory  literatures  (e.g.,  Mathieu  et  al.,  2000, 
Moreland,  1999),  in  studies  of  military  supervisory  systems  and  policies  (e.g.,  Shattuck  & 
Woods,  1997),  and  analysis  of  military  command  and  control  environments  (e.g.,  Oonk  et 
al.,  2000). 

The  results  also  have  implications  for  customization  of  information  products  in  multi¬ 
consumer  environments,  suggesting  that  there  is  a  tradeoff  between  effectiveness  and 
efficiency.  If  the  emphasis  were  only  to  support  the  most  efficient  information  exchange 
in  such  an  environment,  i.e.,  preventing  consumers  from  accessing  irrelevant  information 
at  risk  of  some  consumers  not  getting  everything  they  need,  then  information  producers 
should  be  given  information  about  the  requirements  of  and  tasks  being  performed  by  just 
one  of  the  consumers.  However,  giving  producers  information  about  just  a  single 
consumer  might  be  a  detriment  if  it  is  important  to  have  effective  information  exchange  - 
i.e.,  for  all  consumers  to  get  the  information  they  need.  If  the  emphasis  is  the  support  of 
effective  information  exchange,  then  information  about  all  consumers  should  be 
displayed.  Unfortunately,  given  the  large  number  of  potential  consumers,  this  is  simply 
not  feasible.  Of  course,  practically  speaking  it  is  unlikely  that  the  military  will  ever  focus 
on  efficiency  or  effectiveness  alone;  rather  it  is  more  likely  that  a  balance  will  always 
have  to  be  struck  between  the  two  and  that  a  sensitivity  to  this  issue  will  have  to  be 
addressed  in  future  business  practices  and  technologies  to  minimize  the  effect  highlighted 
by  the  experiment  results. 

An  issue  related  to  customization  of  information  products,  especially  in  hierarchical 
environments  like  the  military,  is  that  of  consumer  “status.”  Participants  in  the 
experiments  biased  their  behavior  towards  the  consumer  needs  of  the  Commander  over 
that  of  his  Deputy,  who  had  a  lower  position  in  the  organization.  This  finding  is  intuitive 
and  consistent  with  others  in  the  literature  (Heacox,  Gwynne,  Kelly,  &  Sander,  2000; 
Torrance,  1955,  as  cited  in  Milanovich,  Driskell,  Stout,  &  Salas,  1998),  where  higher- 
status  team  members  have  been  shown  to  have  more  influence  on  other  team  member’s 
decision  making  than  lower-status  members.  This  tendency  should  be  taken  into 
consideration  when  deciding  what  information  about  consumers  is  communicated  to 
producers. 


The  results  also  have  implications  for  the  design  of  tools  and  displays  that  communicate 
situation  context  in  multi-echelon,  multi-consumer  collaborative  environments  -  and  thus 
promote  shared  understanding  and  synchronized  mental  models.  Our  findings,  however, 
suggest  further  questions  and  issues  exist  with  respect  to  how  to  display  situation  context 
to  information  producers.  These  issues  include: 

•  The  breadth  of  information  that  is  displayed.  It  is  not  practical  to  communicate  all 
missions,  tasks,  and  information  requirements  of  all  consumers  in  most  multi¬ 
consumer  environments  and  doing  so  could  make  the  display  less  usable,  causing 
information  overload.  Several  questions  are  related  to  this  issue,  and  their  answers 
are  likely  dependent  on  the  organization/mission.  They  include: 

1.  Level  of  detail.  How  many  tasks/information  requirements  should  be  displayed? 

2.  Organizational  scope :  How  many  consumers  should  be  included?  Should  there 
be  a  single  display  or  a  separate  one  for  each  consumer? 

3.  Display  organization:  Should  the  displays  be  organized  by  one  of  the  categories 
of  information  provided,  e.g.,  current  tasks,  or  some  other  factor,  such  as 
consumer  or  priority  of  tasks? 

•  Display  format:  Should  the  display  be  text-based  (as  in  the  experiment),  graphical, 
or  a  mixture?  Should  the  display  be  an  “interactive,”  computer-based  interface  or 
static? 

•  Other  features'.  Potential  features  include  highlighting  of  task  and  information 
priorities,  alerting  of  changes  to  new  information  and  providing  other  details  about 
the  consumer. 

These  issues  need  to  be  considered  in  the  design  of  any  “situation  context  display”  and 
should  be  the  focus  of  future  research. 

Follow-on  Work 

Situation  Context  Displays.  Based  on  the  results  of  the  above  experiments,  two  prototype 
displays  that  aimed  at  communicating  situation  context  were  developed.  One 
communicated  this  information  in  a  text-based  interface  and  one  communicated  via  a 
graphical  interface.  They  were  recently  demonstrated  and  evaluated  during  an  applied 
pilot  experiment  conducted  in  the  Multi-Echelon  Command  and  Control  (MECC) 
Laboratory  at  the  Naval  Postgraduate  School.  These  displays  were  used  by  teams  of 
SMEs  as  they  performed  a  collaboration/coordination  task.  Performance  measures 
included  speed  and  accuracy  of  communications,  accuracy  and  relevance  of  exchanged 
information,  and  situation  awareness,  as  well  as  a  variety  of  subjective  measures. 
Although  only  a  pilot  experiment,  the  data  provided  preliminary  suggestions  about 
changes  to  the  prototype  display.  An  example  display  based  on  these  changes  is  shown  in 
Figure  5.  This  display  communicates  information  about  the  mission,  organization, 
consumer  task,  information  requirements  and  feedback  with  new  information  highlighted 
by  diamonds,  hyperlinks  to  more  detail  and  visual  linking  to  a  map  of  the  mission  area. 


Mission:  Conduct  Humanitarian  Assistance  and  Disaster  Relief  operations.  Our  primary  goals  are  to: 

•  Coordinate  with  Philippine  government  /  coalition  partners. 

•  Evacuate  population  from  areas  threatened  bv  violence. 

•  Provide  security  assistance  to  the  current  Philippine  government. 

•  Maintain  status  quo  with  regard  to  insurgent  activities. 


Team 


Team  Lead:  Feedback  and  Information 

MyTasks 

•  Develop  and  maintain  Situation  Awareness 
relevant  to  the  planned  evacuation  of  refugees. 

•  CJTF  has  Identified  Slquljor  as  evacuation  site. 

Mylnformation  Requirements 

•  Known  or  suspected  rebel  units  and  activities. 

•  Christian  refugees. 

•  Evacuation  resources  and  activities 

•  Siquijor  conditions  and  facilities. 

Feedback 

•  |Po4«i  i2/22/)30903|  Team  members,  please  provide 
me  with  more  information  about  that  rebel 
aircraft  we  spotted 


Figure  5.  Example  Situation  Context  Display. 


Changes  in  situation  context.  One  factor  that  has  emerged  from  our  previous  cognitive 
task  analyses  and  research  is  that  both  minor  and  major  changes  to  mission  objectives  or 
producers’  and  consumers’  organizational  structure  directly  impact  those  with  whom  they 
share  information  and  the  kinds  of  information  shared.  Often,  as  change  occurs, 
consumers  have  no  easy  way  to  notify  producers  of  such  changes,  and  no  easy  or 
effective  way  to  provide  feedback  regarding  the  information  that  is  shared.  As  a  result, 
the  overall  efficiency  and  effectiveness  of  information  exchange  suffers.  Future  research 
will  continue  and  extend  our  current  findings,  with  a  focus  placed  on  understanding  the 
effects  of  change  on  information  exchange  (e.g.,  to  mission  goals,  organizational 
structures,  and  consumers’  tasks  and  information  requirements),  and  on  the  development 
of  technologies  and  business  rules  to  help  manage  the  effects  of  such  changes. 
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Overall  Goal:  Enhance  distributed  multi-echelon  knowledge 
exchange  &  command  and  control,  via: 

-  The  identification  of  problems  inherent  in  such  environments. 

-  Furthering  our  understanding  of  the  process  of  information  exchange 

»  Especially,  the  importance  of  shared  task  understanding 

-  Developing  models  and  methods  of  knowledge  transaction  and  tools 
to  facilitate  effective  and  efficient  information  exchange 

Command  21  Program  has  many  facets: 

-  Command  Center  design  (and  supporting  tools  and  business  rules) 

-  Knowledge  Web,  Knowledge  Walls  &  Desks,  and  info  production  and 
knowledge  management  tools 

-  Exploratory  and  applied  research  toward  improving  information 
exchange,  command  &  control,  situation  awareness,  and  decision 
making 


Lessons  Learned 

spa  war  (from  previous  Command  21  Efforts) 
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Knowledge  Web  (K-Web) 


-  Use  at  Global  2000  /  2001  War  Games  revealed  that: 


»  Mission  goals,  Commander’s  Intent,  Commander’s  tasks  and  info 
requirements  were  not  often  shared  effectively  (and  were  not  always 
understood  by  subordinates) 

»  Multi-echelon  aspects  of  these  games  highlighted  disconnect  between 
levels  of  command,  and  between  joint  and  coalition  partners 

-  Use  onboard  USS  Carl  Vinson  (and  others)  confirmed  issues: 

»  Subordinate  commands  that  did  not  have  K-Web  reported 

misunderstandings  regarding  Commander’s  Intent,  Commander’s  tasks 
and  info  requirements,  etc. 

»  Much  higher  (and  lower)  echelons  reported  that  K-Web  content  didn’t 
always  meet  their  needs 


Fleet  Interaction 

-  Past  CTA’s,  observations,  and  ongoing  interaction  validate 
problems  with  sharing  and  shared  understanding  of  context 

Chat  Survey 

-  Recent  survey  Of  fleet  Chat  users  (see  related  CCRTS  paper  and 
presentation)  revealed  significant  issues  associated  with  sharing 
info  in  multi-echelon  /  multi-national  environments 
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Information  Exchange  Environments 
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Information  producers  have  access  to  information  needed  by  information 
consumers.  Their  job  -  information  production  -  is  difficult,  involving: 

1 .  Determining  which  information  is  relevant  and  which  is  not  relevant  to  the 
consumer’s  task  and  goals 

2.  Restating,  reformatting,  integrating  relevant  information,  and  adding  value 


and  creating  new  information  to  create  a  useful,  useable  information 
product. 


Information 

Producer 


Information 

Consumer 
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Effective  and  Efficient 
Information  Exchange 


In  all  collaborative  environments,  optimal 
decision-making  depends  on  information 
exchange  being  as  effective  and  efficient  as 
possible.  We  define  these  as: 

-  Effective  =  the  consumer  gets  all  the  information  he 
needs  to  accomplish  his  tasks 

»  i.e.,  gets  all  the  relevant  information  available 

-  Efficient  =  the  consumer  does  not  get  any 
information  that  distracts  him  from  his  tasks 

»  i.e.,  does  not  gets  any  not  relevant  information 
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Research  Goals 


Identify  and  develop  strategies  and  tools  to 
improve  information  exchange  and  overall 
situation  awareness 

Make  information  exchange  as  effective  and 
efficient  as  possible. 
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Previous  research  underscores  an  important 

requirement,  i.e.,  Communication  of  Situation  Context 
in  information  exchange  environments 

-  Cognitive  task  analyses  and  evaluations 

-  Mental  model  literature 

-  Transactive  memory  literature 

Communication  of  Situation  Context  involves  shared 
understanding  of: 


»  Tasks  and  skills  of  team  members 
»  Information  requirements  of  team  members 
»  Roles  and  role  relationships 
»  Information  flow 

»  Information  sources  and  knowledge 


Research  Questions 
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Obvious  Hypothesis:  Information  exchange  should 
improve  if  producers  have  more  situation  context  - 
know  more  about  consumers. 

However,  questions  remain  about  what  this  information 
should  be: 

-  What  aspects  of  situation  context  are  important? 

-  Are  some  types  information  about  consumers  more  important 
than  others? 

-  What  happens  when  there  are  multiple  consumers? 

-  Does  consumer  “status”  affect  information  exchange? 
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To  measure  information  exchange  efficiency  and 
effectiveness  in  an  experimental  setting  we 
chose  to  examine  producer  performance  at 
determining  relevance  of  information  based  on 
what  they  understand  about  consumer 

-  Our  paradigm: 


»  Participants  (producers)  review  information  items 
»  Shown  situation  context  &  information  about  the  consumer 

»  Given  a  realistic,  limited  amount  of  time  to  sort  the  items  based 
on  whether  they  believe  items  are: 

♦  Relevant  to  the  consumer  ->  Efficiency 

♦  Not  relevant  to  the  consumer  ->  Effectiveness 
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Signal  Detection  theory  was  used  to  measure  performance: 

-  Use  SMEs  to  determine  relevance  of  information  to  consumers 

-  Measure  degree  of  concordance  between  consumer  and  producer 
understanding  of  how  relevant  information  is;  for  example: 

»  Hit:  Producer  and  consumer  both  rate  information  as  relevant 
»  False  alarm:  Producer  rates  information  as  relevant  but  Consumer  rates  it  as  not 


relevant 


Metrics,  based  on  SME  ratings: 

-  Accuracy:  Proportion  correct 

-  Sensitivity  (d5):  Degree  of  overlap  between  producer  /  consumer 
understanding  of  information  relevance 

»  Allows  us  to  examine  how  well  producer  understands  consumer’s  needs 

-  Response  Bias  (P):  Tendency  of  producer  to  sort  information  as 
relevant  or  not  relevant 

»  Allows  us  to  examine  biases  towards  effective  and  efficient  information  exchange 
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Current  Experiments 


Two  related  experiments 
examined  how  to  display 
situation  context  to 
producers 

-  What  does  the  producer  need  to  know 
about  consumer(s)? 

-  What  is  the  effect  of  tailoring 
information  to  a  specific  consumer? 

»  e.g.,  How  useful  is  information  created  for 
consumer  at  one  echelon  to  consumer  at  a 
different  echelon. 
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Participants  played  role  of  producer 
sorted  for  either  one  of  two,  or  both 
consumers  in  a  multi-consumer 
organization. 

-  “Highest”  echelon  =  Commander,  Joint 
Task  Force  (CJTF),  consumer 

-  “Middle”  echelon  =  Deputy  Operations 
Officer  (DOO),  consumer 

-  “Lowest”  echelon  =  producer 
(participant)  creates  products  used  by 

•  CJTF 

•  DOO 

•  Both  CJTF  and  DOO 


who 


12 


What  we  manipulated 
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What  does  the  producer  need  to  know  about 
consumer(s)? 

-  Participants  sorted  information  related  to 
Humanitarian  Assistance  -  Disaster  Relief  scenario 
based  on  information  we  shared  regarding: 


»  Mission  only 
»  Mission  +  consumer’s  tasks 
»  Mission  +  consumer’s  information  requirements 
»  Mission  +  consumer’s  tasks  and  information  requirements 


-  Results  allowed  us  to  determine  which  aspects  of 
situation  context  are  most  important  to 
communicate  to  producers 
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What  we  manipulated...  (2) 


What  is  the  effect  of  tailoring  information  to  a 

specific  consumer  or  multiple  consumers? 

-  In  one  experiment,  participants  sorted 
information  for  only  one  of  the  two 
consumers. 

->  This  allowed  us  to  look  at  the  cost  to  the 
other  consumer  if  they  needed  to  use  the 
same  information  product 

-  In  the  other  experiment,  participants  sorted 
information  for  only  one  of  the  two 
consumers,  and  then  for  both  consumers  (in 
a  separate  block  /  run  /  trial) 

->  This  allowed  us  to  look  at  the  how  useful  an 
information  product  created  for  multiple 
<r^consumers  was  to  each. 
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Results 


SPA  WAR 
Systems  Center 
San  Diego 


Effectiveness  of  Information  Exchange 


Q:  What  aspects  of  situation  context  should  be 

communicated  to  producers  to  improve  effectiveness? 

(Effectiveness  =  performance  at  sorting  relevant  info) 

A:  Highest  accuracy  and  sensitivity  when  participants 
given  consumer  task  and  information  requirements 
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JL  Efficiency  of  Information  Exchange 

Systems  Center 
San  Diego 


Q:  What  aspects  of  situation  context  should  be 
communicated  to  producers  to  improve  efficiency? 

(Efficiency  =  performance  at  sorting  not  relevant  info) 

A:  Highest  accuracy  and  sensitivity  when  participants 
given  consumer  task  and  information  requirements 
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What  aspects  of  situation  context 
should  be  communicated? 


So  far,  results  suggest.... 

-  Consumers’  mission,  task,  and  information 
requirements  are  all  important  and  necessary 
aspects  of  situation  context  and  therefore  must  be 
communicated  to  producers 

-  These  results  highlight  the  importance  of 
maintaining  a  shared  situation  context 


No  real  surprises  here . . . 
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But  the  real  world  is  not  so 
simple... 


What  about  in  multi-consumer 
situations? 

(which  are  the  norm...) 

-  When  participants  sorted  for  one 
specific  consumer 

»  There  was  a  cost  when  other 
consumers,  with  different  tasks  and 
requirements,  are  required  to  use 
information  products  produced  for 
someone  else 

»  Especially  with  respect  to  effectiveness 
of  information  exchange 

»  The  more  relevant  the  information  was 
for  one  consumer,  the  more  not  relevant 
it  often  was  for  other  consumers... 
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Multiple  consumers  (cont’d) 


When  participants  sorted  for 
both  consumers 

-  There  was  a  bias  in 
selecting  information  to 
share 

»  Producers  erred  on  the  side  of 
providing  too  much  information 
rather  than  risk  not  providing 
enough 
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Does  consumer  “status”  affect 
information  exchange? 


There  was  a  bias  towards  meeting  needs  of 
higher-level  consumer 

-  Overall  performance  was  lower  when  participant 
was  required  to  sort  for  two  consumers  (even  if  one 
was  of  higher  status) 

-  Overall  performance  was  better  when  CJTF  was  the 
only  consumer  (users  worked  harder  -  were  more 
effective  and  efficient  -  when  someone  of  very  high 
rank  was  their  only  customer) 
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To  make  information  producers  more 
effective  and  efficient: 

-  Explicitly  communicate  mission,  task,  and 
information  requirements 

-  Include  information  about  all  potential  consumers 

»  Difficult  to  implement  when  number  of  consumers  is  large... 

-  Recognize  bias  towards  consumers  with  higher 
status 

»  Develop  and  follow  business  rules  to  moderate 

-  Incorporate  other  aspects  of  situation  context  in 
situation  context  displays 

»  Future  research  will  examine  effects  of  changes  to  the  situation 


22 


SPA  WAR 
Systems  Center 
San  Diego 


Early  Application  of  Results 
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These  results  have  led  to: 

-  New  business  rules  and  recommendations 

-  Development  of  a  prototype  Consumer  Information 
Requirements  Display  Tool  (templates)  to  support 
information  exchange  in  multi-echelon 
environments 

»  Demonstrated  at  MECC  Lab  (at  Naval  Postgraduate  School)  using 
SMEs  in  a  multi-echelon,  distributed  teams  environment 
performing  a  mission  that  required  extensive  information  sharing 

»  Based  on  this  applied  work...  (See  next  slide) 
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Notional  Situation  Context  Display 
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Mission:  Conduct  Humanitarian  Assistance  and  Disaster  Relief  operations.  Our  primary  goals  are: 

•  Coordinate  with  Philippine  government  /  coalition  partners. 

•  Evacuate  population  from  areas  threatened  by  violence. 

•  Provide  security  assistance  to  the  current  Philippine  government. 

•  Maintain  status  quo  with  regard  to  insurgent  activities. 
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Battle  Watch  Captain 
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Tactical  Watch  Officer 


Team  Lead:  Feedback  and  Information 

My  Tasks 

•  Develop  and  maintain  Situation  Awareness 
relevant  to  the  planned  evacuation  of  refugees. 

•  CJTF  has  identified  Siquiior  as  evacuation  site. 

My  Information  Requirements 

•  Known  or  suspected  rebel  units  and  activities. 

•  Christian  refugees. 

•  Evacuation  resources  and  activities. 

•  Siquiior  conditions  and  facilities. 


Feedback 

■  [Posted  12/22/030903]  Team  members,  please  provide 
me  with  more  information  about  that  rebel 
aircraft  we  spotted 
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Questions? 
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